Skip to content

Track and report network stat #124

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom
Draft
wants to merge 3 commits into
base: main
Choose a base branch
from
Draft

Track and report network stat #124

wants to merge 3 commits into from

Conversation

@seanses
Copy link
Collaborator

@seanses seanses commented Dec 18, 2024 •
edited
Loading

In progress adding number of retries to the metrics.

Example tracing:

{"timestamp":"2024-12-18T10:04:56.945534Z","level":"INFO","fields":{"message":"Xorb upload accumulated rate: 74.54 Mbps"},"filename":"/home/ubuntu/data/xet-core/data/src/parallel_xorb_uploader.rs","line_number":90}
{"timestamp":"2024-12-18T10:04:56.945589Z","level":"INFO","fields":{"message":"Xorb upload instantaneous rate: 74.54 Mbps"},"filename":"/home/ubuntu/data/xet-core/data/src/parallel_xorb_uploader.rs","line_number":90}
{"timestamp":"2024-12-18T10:04:59.166637Z","level":"INFO","fields":{"message":"Xorb upload accumulated rate: 284.91 Mbps"},"filename":"/home/ubuntu/data/xet-core/data/src/parallel_xorb_uploader.rs","line_number":90}
{"timestamp":"2024-12-18T10:04:59.166665Z","level":"INFO","fields":{"message":"Xorb upload instantaneous rate: 966.45 Mbps"},"filename":"/home/ubuntu/data/xet-core/data/src/parallel_xorb_uploader.rs","line_number":90}
...
{"timestamp":"2024-12-18T10:05:14.036182Z","level":"INFO","fields":{"message":"Xorb upload instantaneous rate: 2.62 Gbps"},"filename":"/home/ubuntu/data/xet-core/data/src/parallel_xorb_uploader.rs","line_number":90}
{"timestamp":"2024-12-18T10:05:16.244455Z","level":"INFO","fields":{"message":"Xorb upload accumulated rate: 1.32 Gbps"},"filename":"/home/ubuntu/data/xet-core/data/src/parallel_xorb_uploader.rs","line_number":90}
{"timestamp":"2024-12-18T10:05:16.244487Z","level":"INFO","fields":{"message":"Xorb upload instantaneous rate: 2.19 Gbps"},"filename":"/home/ubuntu/data/xet-core/data/src/parallel_xorb_uploader.rs","line_number":90}

@seanses seanses marked this pull request as draft December 18, 2024 10:11
@seanses seanses requested a review from port8080 December 18, 2024 10:17
@seanses seanses changed the title Track and report egress rate Track and report network stat Dec 18, 2024
port8080
port8080 reviewed Dec 18, 2024
View reviewed changes
data/src/parallel_xorb_uploader.rs
type XorbUploadValueType = (MerkleHash, Vec<u8>, Vec<(MerkleHash, usize)>);
struct NetworkStatCheckPoint {
n_bytes: u64,
start: Instant,
Copy link

@port8080 port8080 Dec 18, 2024

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I found using std::time::SystemTime more useful in sending these over compared to Instant in #121

port8080
port8080 reviewed Dec 18, 2024
View reviewed changes
data/src/parallel_xorb_uploader.rs
use crate::errors::DataProcessingError::*;
use crate::errors::*;

const DEFAULT_NETWORK_STAT_REPORT_INTERVAL_SEC: u32 = 2; // 2 s
Copy link

@port8080 port8080 Dec 18, 2024 •
edited
Loading

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

You probably have this set low for testing, would prefer larger in prod - maybe 100s

port8080
port8080 reviewed Dec 18, 2024
View reviewed changes
data/src/parallel_xorb_uploader.rs
while let Some(result) = upload_tasks.join_next().await {
result??;
let metrics = result??;
let mut egress_rate = self.egress_stat.lock().await;
Copy link

@port8080 port8080 Dec 18, 2024

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Try the lock and optimistically report. I think it is completely OK to skip the update_and_report call if there is lock contention

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Reviewers

1 more reviewer

@port8080 port8080 port8080 left review comments

Reviewers whose approvals may not affect merge requirements

At least 1 approving review is required to merge this pull request.

Assignees

No one assigned

Labels

None yet

Projects

None yet

Milestone

No milestone

Development

Successfully merging this pull request may close these issues.

3 participants

@seanses @port8080